Information processing apparatus, information processing method, and program based on operator probability using sensors

ABSTRACT

The present invention provides an information processing apparatus for carrying out processes in response to how a user behaves. The information processing apparatus includes a sensor device, operator probability computation means, and processing means. The sensor device is configured to sense conditions. The operator probability computation means is configured to compute an operator probability representing how likely each of at least one target is to be an operator operating the information processing apparatus on the basis of an output of the sensor device, the target being the user staying close to the apparatus. The processing means configured to perform a predetermined process based on the operator probability.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese PatentApplication JP 2006138586 filed with the Japanese Patent Office on May18, 2006, the entire contents of which being incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an information processing apparatus, aninformation processing method, and a program. More particularly, theinvention relates to an information processing apparatus, an informationprocessing method, and a program for offering relevant services to userswithout requiring them to carry out elaborate actions.

2. Description of the Related Art

Illustratively, there exist so-called interaction apparatuses capable ofperforming diverse processes through interaction with users. Theinteraction apparatus typically accepts users' actions such as speechesand gestures as interaction signals and executes processes correspondingto these interaction signals. In this manner, the interaction apparatusmay recommend TV programs, change channels, and perform other actionsfor the users' sake in response to their actions.

In the description that follows, the users who stay near the interactionapparatus will be referred to as targets. Likewise the person whooperates (or is willing to operate) the interaction apparatus will becalled the operator where appropriate.

There exist techniques for identifying gestures of people based on theirimages picked up by an image sensor, thereby estimating their actionsand intentions (e.g., see Japanese Patent Laid-open No. 2005-242759).

There have also been proposed interface devices that identify personsand recognize their gestures so as to control the equipment (e.g., seeInternational Publication WO 2003/025859 Pamphlet).

SUMMARY OF THE INVENTION

Some of the interaction apparatuses proposed so far assume that there isone target near the apparatus and that the apparatus performs actions bytaking the single target for the operator.

This type of interaction apparatus is typically designed to assume anarrow space resembling that of a telephone booth as the space in whichone target can reside. The intention is to uphold the assumption thatthere is only target.

There may well be a plurality of targets coming into that narrow space.In anticipation of such cases, the interaction apparatus is typicallyconstrained to select one of the multiple targets as the operator on thebasis of heuristic rules devised empirically by designers (e.g., onerule may assume that a person positioned far away is not the operator;another may assume that a person just passing by is not the operator).

Another type of the existing interaction apparatuses assumes that thereare a plurality of targets near the apparatus and that any one of themcan be the operator. In this case, the action of each of the multipletargets is sensed and accepted as an interaction signal by theapparatus.

However, where the interaction apparatus takes the action of each of aplurality of targets to be an interaction signal, the true operatorwishing to actually operate the interaction apparatus is often disruptedby the actions of the other targets.

Generally, the users who could each be a potential target often stayclose to the interaction apparatus without being aware of it. As aresult, the interaction apparatus often recognizes mistakenly thespeeches and gestures of those users with no intention of becomingoperators as interaction signals that will hamper the apparatus fromgetting operated by the user willing to become the operator.

In order to bypass such problems, the interaction apparatus may bearranged to select a particular user as the target if that user gives aspecific speech such as “I am the user” or makes a predeterminedgesture.

However, the arrangement requiring the user to give an explicit speech(e.g., “I am the user”) or carry out a particular gesture forces theuser to execute complicated actions preparatory to starting theinteraction apparatus. This makes it difficult to start utilizing theinteraction apparatus smoothly and quickly, inconveniencing the user.

Illustratively, suppose that the interaction apparatus has the abilityto recommend TV programs matching a given operator's preferences andthat a target wants the apparatus to recommend TV programs to him orher. In that case, it is necessary for any one target to manifesthimself or herself as the operator by performing an explicit action suchas saying “I am the user.” If two targets are near the interactionapparatus and if one of them has been acting as the operator, the othertarget performs an explicit action when expressing his or her intentionto become the operator (e.g., saying “I am the user”), which can be avery complicated procedure.

In order to anticipate any target carrying out an explicit action toexpress his or her intention to become the operator, the interactionapparatus prepares for interrupt handling throughout its interactionprocessing. To implement such processing needs elaborate softwaredesign.

The interaction apparatus recommending TV programs matching theoperator's preferences as described above stores beforehand personalinformation including the preferences of the targets who are operatorcandidates. One of the targets near the interaction apparatus isselected as the operator and recognized (identified) as such throughsensing techniques such as face recognition. TV programs are thenrecommended on the basis of the preference information about theoperator thus recognized.

Some of the existing interaction apparatuses recognize each of aplurality of targets nearby as operators and accept all of their actionsas interaction signals as outlined above. This type of interactionapparatus is capable of recognizing multiple targets through facerecognition, but has difficulty in providing services such asrecommendation of TV programs by suitably utilizing the preferenceinformation about each of the targets.

The typical interaction apparatus accepting the actions of all targetsas interaction signals can generally perform one of two things: afterrecognizing each of multiple targets through face recognition, theapparatus may pick up one of them as the main operator and recommend TVprograms based on the preference information about that operator; or theinteraction apparatus may average the preference information about alltargets and recommend TV programs based on the averaged preferenceinformation.

Still another type of interaction apparatus may conceivably need each ofa plurality of targets nearby to enter through a keyboard or the likehis or her personal information and likelihood of becoming the operator.This arrangement would make it possible to recommend TV programs byutilizing the preference information about each of the multiple targetsin a manner reflecting his or her likelihood of becoming the operator.

However, the arrangement above is laborious in that it involves havingeach target enter his or her personal information as well as his or herlikelihood of becoming the operator into the interaction apparatus. Inthe event of added or reduced targets nearby, the interaction apparatusis incapable of quickly addressing the changes. This can be asignificant inconvenience for the users.

The need has been felt for an interaction apparatus capable of offeringrelevant services to users as targets without requiring them to performonerous actions such as an explicit declaration of being the operator.

The present invention has been made in view of the above circumstancesand provides arrangements whereby users are offered suitable serviceswithout having to perform complicated actions.

In carrying out the present invention and according to one embodimentthereof, an information processing apparatus for carrying out processesin response to how a user behaves includes: a sensor device configuredto sense conditions; operator probability computation means configuredto compute an operator probability representing how likely each of atleast one target is to be an operator operating the informationprocessing apparatus on the basis of an output of the sensor device, thetarget being the user staying close to the apparatus; and processingmeans configured to perform a predetermined process based on theoperator probability.

According to another embodiment of the present invention, an informationprocessing method or a program for causing an information processingapparatus or a program to carry out an information processing procedurein response to how a user behaves includes the steps of: computing anoperator probability representing how likely each of at least one targetis to be an operator operating the information processing apparatus onthe basis of an output of a sensor device configured to senseconditions, the target being the user staying close to the apparatus;and performing a predetermined process based on the operatorprobability.

Where the information processing apparatus, information processingmethod, or program according to the present invention is in use, theprobability of each of at least one target being an operator operatingthe apparatus or the computer is computed as the operator probability onthe basis of an output of a sensor device configured to senseconditions, the target being the user staying close to the apparatus orthe computer. A predetermined process is then performed on the basis ofthe operator probability thus computed.

According the above-outlined embodiments of the present invention, theuser may be offered relevant services without having to behave in anelaborate or complicated manner.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a typical structure of an AV systemaccording to the embodiment of the present invention;

FIG. 2 is a flowchart of steps constituting a process performed by aninteraction device;

FIG. 3 is a flowchart of steps constituting a process performed by aprogram recommendation device;

FIG. 4 is a tabular view showing a preference table;

FIG. 5 is a schematic view showing a recommendation screen;

FIG. 6 is a block diagram showing a typical structure of a targetinformation acquisition device;

FIG. 7 is a schematic view explanatory of processing by an imageprocessing device;

FIG. 8 is another schematic view explanatory of the processing by theimage processing device;

FIG. 9 is another schematic view explanatory of the processing by theimage processing device;

FIG. 10 is a schematic view explanatory of processing by a targetinformation creation device;

FIG. 11 is a schematic view showing what is contained in a targetinformation storage device;

FIG. 12 is a schematic view showing a Bayesian network (BN) model usedto compute operator probabilities;

FIG. 13 is a schematic view showing typical structures of a learningapparatus for acquiring a network structure and a conditionalprobability table (CPT) through learning;

FIG. 14 is a schematic view showing typical operator probabilities forthe targets being imaged;

FIG. 15 is a flowchart of steps constituting a process performed by anoperator probability computation device;

FIG. 16 is a schematic view explanatory of an outline of what a harddisk (HD) recorder does; and

FIG. 17 is a block diagram showing a typical structure of a computeraccording to the embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

What is described below as the preferred embodiments of the presentinvention with reference to the accompanying drawings corresponds to theappended claims as follows: the description of the preferred embodimentsbasically provides specific examples supporting what is claimed. If anyexample of the invention described below as a preferred embodiment doesnot have an exactly corresponding claim, this does not means that theexample in question has no relevance to the claims. Conversely, if anyexample of the invention described hereunder has a specificallycorresponding claim, this does not mean that the example in question islimited to that claim or has no relevance to other claims.

One embodiment of the present invention is an information processingapparatus (e.g., HD recorder 3 in FIG. 1) for carrying cut processes inresponse to how a user behaves. The information processing apparatusincludes: a sensor device (e.g., sensor device 41 in FIG. 1) configuredto sense conditions; operator probability computation means (e.g.,operator probability computation device 42 in FIG. 1) configured tocompute an operator probability representing how likely each of at leastone target is to be an operator operating the information processingapparatus on the basis of an output of the sensor device, the targetbeing the user staying close to the apparatus; and processing means(e.g., interaction device 14 and program recommendation device 15 inFIG. 1) configured to perform a predetermined process based on theoperator probability.

Preferably, the operator probability computation means may furtherinclude: information extraction means (e.g., information extractiondevice 101 in FIG. 6) configured to extract from the output of thesensor device, information about each of such at least one target;target information creation means (e.g., target information creationdevice 102 in FIG. 6) configured to create target information about eachof such at least one target out of the information extracted by theinformation extraction means; and probability computation means (e.g.,probability computation device 104 in FIG. 6) configured to compute theoperator probability for each of such at least one target on the basisof the target information about each of such at least one target.

Preferably, the operator probability computation means may furtherinclude storage means (e.g., target information storage device 103 inFIG. 6) configured to store the target information createdchronologically by the target information creation means; and the proability computation means may compute the operator probability based onthe chronological target information stored in the storage means.

Preferably, the processing means may further include: operator selectionmeans (e.g., operator selection device 51 in FIG. 1) configured toselect the target with the highest operator probability as the operator;action selection means (e.g., action selection device 52 in FIG. 1)configured to select an action to be performed in response to how theoperator behaves; and action execution means (e.g., action executiondevice 53 in FIG. 1) configured to execute a process corresponding tothe action selected by the action selection means.

Preferably, the processing means may further include: preferenceinformation storage means (e.g., preference table storage device 61 inFIG. 1) configured to store preference information representative ofpreferences of each of such at least one target; preference informationprocessing means (e.g., preference information processing device 62 inFIG. 1) configured to create program recommendation-intended preferenceinformation for recommending programs out of the preference informationabout each of such at least one target in accordance with the operatorprobability for each of such at least one target; and service offeringmeans (e.g., service offering device 63 in FIG. 1) configured to offer aprogram recommending service based on the programrecommendation-intended preference information.

Another embodiment of the present invention is an information processingmethod or a program for causing an information processing apparatus or acomputer to carry out an information processing procedure in response tohow a user behaves. The procedure includes the steps of: computing(e.g., in steps S101 through S104 of FIG. 15) an operator probabilityrepresenting how likely each of at least one target is to be an operatoroperating the information processing apparatus or the commuter on thebasis of an output of a sensor device configured to sense conditions,the target being the user staying close to the apparatus or thecomputer; and performing (e.g., in steps S11 through S14 of FIG. 2 andin steps S21 through S24 of FIG. 3) a predetermined process based on theoperator probability.

The preferred embodiments of the present invention will now be describedin more detail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a typical structure of an AV (AudioVisual) system acting as an interaction apparatus according to theembodiment of the present invention.

In FIG. 1, the AV system is made up of a display device 1, speakers 2,and a hard disk (HD) recorder 3.

The display device 1 is constituted illustratively by a CRT (Cathode RayTube) or an LCD (Liquid Crystal Display) that displays imagescorresponding to video data supplied by the HD recorder 3. The speakers2 output sounds that reflect audio data fed by the HD recorder 3.

The HD recorder 3 is composed of a recording/reproduction device 11, aTV (television) device 12, a target information acquisition device 13,an interaction device 14, and a program recommendation device 15.

The recording/reproduction device 11 is formed by arecording/reproduction control device 21 and a recording medium 22. Thusstructured, the recording/reproduction device 11 records and reproducesvideo and audio data.

More specifically, the recording/reproduction device 21 records to therecording medium 22 video and audio data supplied by the TV device 12.From the recording medium 22, the recording/reproduction device 21reproduces (i.e., reads) video and audio data and sends the retrieveddata to the TV device 12.

The recording medium 22, typically a hard disk, is driven by therecording/reproduction control device 21 to record video and audio data.

Alternatively, the recording/reproduction control device 21 may writeand read video and audio data to and from a recording medium such as aDVD (Digital Versatile Disc), not shown, that can be removably attachedto the HD recorder 3.

The TV device 12 is made up of a program reception device 31, a signalprocessing device 32, an EPG (Electronic Program Guide) storage device33, and an output control device 34. The TV device 12 acquires TVprogram data and other information and feeds what is acquired to thedisplay device 1, speakers 2, and recording/reproduction device 11.

Illustratively, the program reception device 31 receives TV broadcastsignals transmitted terrestrially or through a CATV (cable television)setup and supplies the received signals to the signal processing device32. Alternatively, the program reception device 31 may receive TVprogram data and other information from servers on the Internet and feedwhat is received to the signal processing device 32.

The signal processing device 32 suitably processes the data (signals)fed from the program reception device 31 and extracts video and audiodata constituting EPG data ant TV programs from the processed data.Furthermore, the signal processing device 32 supplies the EPG data tothe EPG storage device 33 and sends the video and audio dataconstituting TV programs to the output control device 34.

The EPG storage device 33 stores the EPG data sent from the signalprocessing device 32. Every time new EPG data is supplied from thesignal processing device 32, the EPG storage device 33 updates itsstored contents with the new data. In this manner, the EPG storagedevice 33 retains the latest EPG data. The EPG data held in the EPGstorage device 33 is referenced as necessary by relevant blocks.

The output control device 34 sends the video data supplied from therecording/reproduction device 11 and signal processing device 32 to thedisplay device 1 for display of corresponding images. The output controldevice 34 also feeds the audio data from the same sources to thespeakers 2 for output of corresponding sounds.

The output control device 34 sends the video and audio data coming fromthe signal processing device 32 to the recording/reproduction device 11for recording therein. Furthermore, the output control device 34 readsEPG data from the EPG storage device 33 and supplies the retrieved datato the display device 1 for display of corresponding EPG screens.

In addition, the output control device 34 sends data supplied by otherblocks from time to time to the display device 1 or speakers 2 foroutput of corresponding images or sounds.

The target information acquisition device 13 is formed by a sensordevice 41 and an operator probability computation device 42. Inoperation, the target information acquisition device 13 acquires targetinformation about each of at least one target near the HD recorder 3,the target being the user. The target information thus acquired issupplied to the interaction device 14, program recommendation device 15,and other relevant blocks.

The sensor device 41 senses conditions and outputs sensor signalsreflecting the results of the sensing.

Given the sensor signals from the sensor device 41, the operatorprobability computation device 42 acquires target information about eachof at least one target near the HD recorder 3, the informationrepresenting the behavior of each target (i.e., target conditions). Alsoon the basis of the sensor signals from the sensor device 41, theoperator probability computation device 42 computes an operatorprobability representing how likely each of at least one target near theHD recorder 3 is to be the operator. The operator probability thuscomputed is included in the target information about each target andsupplied to the interaction device 14, program recommendation device 15,and other relevant blocks.

The interaction device 14 is constituted by an operator selection device51, an action selection device 52, and an action execution device 53. Inoperation, the interaction device 14 performs predetermined processesbased on the operator probabilities and other data included in thetarget information sent from the target information acquisition device13.

The operator selection device 51 selects the target with the highestoperator probability included in the target information coming from thetarget information acquisition device 13. The information about theoperator thus selected is forwarded by the operator selection device 51to the action selection device 52.

Given the target information from the operator selection device 51(i.e., information about the operator's behavior), the action selectiondevice 52 selects the action to be performed accordingly, and sendsaction information denoting the selected action to the action executiondevice 53.

The action execution device 53 carries out processes corresponding tothe action represented by the action information coming from the actionselection device 52, i.e., corresponding to the action selected by theaction selection device 52.

The program recommendation device 15 is made up of a preference tablestorage device 61, a preference information processing device 62, and aservice offering device 63. In operation, the program recommendationdevice 15 carries cut predetermined processes based on the operatorprobabilities and other data included in the target information suppliedby the target information acquisition device 13.

The preference table storage device 61 stores a preference table thatstores preference information representing the preferences of each userwho could be the potential target.

The preference information processing device 62 creates preferenceinformation based illustratively on the TV programs viewed by users whocould potentially be the target each and on the TV programs reserved bythese users for unattended recording (i.e., to the recording medium 22).The preference information thus created is written to the preferencetable held by the reference table storage device 61. The TV programsviewed so far by the users include those viewed in real time as well asthose reserved for unattended recording to the recording medium 22before they were reproduced subsequently for viewing.

The preference information processing device 62 creates programrecommendation-intended preference information for recommending TVprograms on the basis of the operator probabilities included in thetarget information coming from the target information acquisition device13 and in accordance with the preference information about the targets(i.e., users) stored in the preference table held by the preferencetable storage device 61. The program recommendation-intended preferenceinformation thus created is supplied to the service offering device 63.

The service offering device 63 offers services recommending TV programsbased on the program recommendation-intended preference informationsupplied from the preference information processing device 62.

More specifically, the service offering device 63 creates arecommendation screen including recommended TV program listings andsends the created screen to the output control device 34 in the TVdevice 12 for display on the display device 1 or for output through thespeakers 2.

Described below with reference to the flowchart of FIG. 2 is a processperformed by the interaction device 14 shown in FIG. 1.

As discussed above, the target information acquisition device 13acquires target information about each of at least one target near theHD recorder 3 and feeds the acquired information to the interactiondevice 14. The operator selection device 51 in the interaction device 14waits for the target information about each target to come from thetarget information acquisition device 13. In step S11, the operatorselection device 51 receives the target information about each targetfrom the target information acquisition device 13. Following receipt ofthe target information, step S12 is reached.

In step S12, the operator selection device 51 selects the target withthe highest operator probability as the operator out of the targetinformation received from the target information acquisition device 13.The selected target information is supplied to the action selectiondevice 52, before step S13 is reached.

In step S13, the action selection device 52 selects the action to becarried out by the HD recorder 3 in response to the operator's behavior,i.e., conditions indicated by the target information coming from theoperator selection device 51. The action selection device 52 sendsaction information denoting the selected action to the action executiondevice 53, before step S14 is reached.

The action execution device 53 executes processes corresponding to theaction denoted by the action information sent from the action selectiondevice 52.

More specifically, if the action information represents the action ofchanging TV programs to be received or of switching channels, then theaction execution device 53 causes the program reception device 31 in theTV set 12 to change the TV programs to be received or to switch thechannels.

Thereafter, the interaction device 14 returns to step S11 when thetarget information about each target is again supplied from the targetinformation acquisition device 13. The subsequent steps are thenrepeated.

As described, the interaction device 14 illustratively selects thetarget with the highest operator probability as the operator. Inresponse to the target information about the selected operator (i.e., inregard to the operator's behavior), the interaction device 14 selectsand executes the action to be carried out by the HD recorder 3.

All targets other than the target with the highest operator probability(i.e., targets with lower operator probabilities) are ignored. As aresult, the likelihood of the behavior of the lesser targets beingmistakenly recognized as that of the operator is significantly reduced.This makes it possible for the HD recorder 3 to execute actions in astable manner for interaction with the user acting as the operator.

There is no need for the user acting as the operator to displaycomplicated or elaborate behavior such as performing an explicit actionmanifesting himself or herself as the operator. Regardless of theabsence of such user intervention, the HD recorder 3 can offer the userrelevant services such as changing of channels as desired by the user.

Described below with reference to the flowchart of FIG. 3 is a processperformed by the program recommendation device 15 shown in FIG. 1.

As described above, the target information acquisition device 13acquires target information about each of at least one target near theHD recorder 3 and forwards the acquired information to the programrecommendation device 15. The reference information processing device 62in the program recommendation device 15 waits for the target informationabout each target to come from the target information acquisition device13. Upon receipt of the target-specific target information from thetarget information acquisition device 13, the preference informationprocessing device 62 goes to step S21. Step S21 is followed by step S22.

In step S22, the preference information processing device 62 createsprogram recommendation-intended preference information for recommendingTV programs on the basis of the operator probabilities included in thetarget information coming from the target information acquisition device13 and in accordance with the preference information about the targets(i.e., users) stored in the preference table held by the preferencetable storage device 61. The program recommendation-intended preferenceinformation thus created is supplied to the service offering device 63.Step S22 is followed by steps S23 and S24, in that order.

In steps S23 and S24, the service offering device 63 offers servicesrecommending TV programs eased on the program recommendation-intendedpreference information supplied from the preference informationprocessing device 62.

More specifically, the service offering device 63 in step S23 determinesat least one TV program to be recommended to the target near the HDrecorder 3 on the basis of both the program recommendation-intendedpreference information from the preference information processing device62 and the EPG data held by the EPG storage device 33 in the TV device12, before step S24 is reached.

In step S24, the service offering device 63 recommends the TV program orprograms determined in step S23. That is, the service offering device 63in step S24 illustratively creates a recommendation screen made up ofthe TV program listings determined to be recommended to the target instep S23. The recommendation screen thus created is sent to the outputcontrol device 34 for display on the display device 1.

Thereafter, the program recommendation device 15 returns to step S21when the target information about each target is again supplied from thetarget information acquisition device 13. The subsequent steps are thenrepeated.

Frequent display of the recommendation screen on the display device 1can disrupt the user's viewing of TV programs on the same device. Thisbottleneck can be bypassed by making arrangements for the recommendationscreen to be displayed if recommendation screen display mode is ineffect or if the user specifically requests display of therecommendation screen.

The program recommendation device 15 shown in FIG. 1, like theinteraction device 14 in the same figure, may select the target with thehighest operator probability as the operator and recommend at least oneTV program matching the preference information about the selectedoperator, the information being used as program recommendation-intendedpreference information.

It is also possible for the program recommendation device 15 in FIG. 1to create program recommendation-intended preference information out ofthe preference information about a plurality of targets and to recommendTV programs matching the program recommendation-intended preferenceinformation thus created.

Described below with reference to FIGS. 4 and 5 are two methods forrecommending TV programs that match the program recommendation-intendedpreference information created out of the preference information aboutmultiple targets.

The two methods to be discussed below are preceded by theabove-described method whereby the target with the highest operatorprobability is selected as the operator; whereby the selected operator'spreference information is determined to be program recommendationintended preference information; and whereby at least one TV programmatching the program recommendation-intended preference information thusdetermined is recommended. This method will be called the firstrecommendation method in the description that follows. Of the twomethods about to be described involving creation of programrecommendation-intended preference information out of the preferenceinformation about multiple targets and recommendation of TV programsmatching the information thus created, one will be referred to as thesecond method and the other as the third method in the ensuingdescription.

FIG. 4 is a tabular view showing a preference table stored in thepreference table storage device 61 of the program recommendation device15 in FIG. 1.

In the preference table of FIG. 4, user labels T_(i) (i=1, 2, . . . , N)identifying N users as potential targets are associated with preferenceinformation v_(i) representing the preferences of the users T_(i)(identified by the user labels T_(i)).

In FIG. 4, the preference information v_(i) is denoted by vectors havingcomponents representative of each user's varying degrees of preferencefor different categories (genres) of TV programs.

For purpose of simplification and illustration, it is assumed that thereare N targets (i.e., users) T₁, T₂, . . . , T_(N) near the HD recorder3. In this case, the target information acquisition device 13 suppliesthe program recommendation device 15 with target information about eachof the N targets T_(i), the target information including the operatorprobability P_(i) representing how likely each of the targets T_(i) isto be the operator.

According to the second recommendation method, the preferenceinformation processing device 62 in the program recommendation device 15synthesizes program recommendation-intended preference information fromthe preference information v_(i) about N targets T_(i) on the basis ofthe operator probabilities P_(i) included in the target informationabout these N targets T_(i).

More specifically, the preference information processing device 62weights the preference information v₁, v₂, . . . v_(N) about the Ntargets T₁, T₂, . . . , T_(N) with the operator probabilities P₁, P₂, .. . , P_(N) for the N targets T₁, T₂, . . . , T_(N). The weightedresults (P₁×v₁+P₂×v₂+ . . . P_(N)×v_(N)) are supplied to the serviceoffering device 63 as program recommendation-intended preferenceinformation.

At this point, the service offering device 63 determines a plurality ofTV programs matching the program recommendation-intended preferenceinformation from the preference information processing device 62 byreferencing the EPG data stored in the EPG storage device 33. Theservice offering device 63 then recommends the determined TV programsillustratively by displaying on the display device 1 a recommendationscreen containing recommended program information such as program titlesabout the multiple TV programs determined.

According to the second recommendation method described above, TVprograms matching the preferences of the targets with higher operatorprobabilities are more likely to be recommended.

According to the third recommendation method, the preference informationprocessing device 62 creates program recommendation-intended preferenceinformation that combines the preference information v_(i) about targetsT_(i) with the number of TV programs recommended on the basis of thatpreference information v_(i) (the number may be called therecommendation count where appropriate). The targets T_(i) are selectedas those having the M highest operator probabilities P_(i) out of Ntargets T_(i) (N≦N) whose target information is supplied from the targetinformation acquisition device 13.

Illustratively, it is assumed that the number M is “3,” that targetsT_(x), T_(y), and T_(z) have the three highest operator probabilitiesP_(x), P_(y), and P_(z) respectively, and that the targets T_(x), T_(y),and T_(z) are assigned preference information v_(x), v_(y), and v_(z)respectively. It is also assumed that up to a total of “k” TV programscan be listed as recommended program information on the recommendationscreen.

In the case above, the preference information processing device 62computes the value corresponding to the operator probability P_(x) forthe target T_(x) as the number of TV programs to be recommended on thebasis of the preference information v_(x) about that target T_(x). Morespecifically, the preference information processing device 62 computesthe recommendation count C_(x)=P_(x)×k/(P_(x)+P_(y)+P_(z)) (fractions ofC_(x) are discarded) proportional to the operator probability P_(x) forthe target T_(x), and has the computed recommendation count C_(x)combined with the preference information v_(x).

Likewise, as the number of TV programs to be recommended on the basis ofthe preference information v_(y) about the target T_(y), the preferenceinformation processing device 62 computes the recommendation countC_(y)=P_(y)×k/(P_(x)+P_(y)+P_(z)) proportional to the operatorprobability P_(y) for the target T_(y), and has the computedrecommendation count C_(y) combined with the preference informationv_(y). As the number of TV programs to be recommended on the basis ofthe preference information v_(z) about the target T_(z), the preferenceinformation processing device 62 computes the recommendation countC_(z)=P_(z)×k/(P_(x)+P_(y)+P_(z)) proportional to the operatorprobability P_(z) for the target T_(z), and has the computedrecommendation count C_(z) combined with the preference informationv_(z).

The preference information processing device 62 then supplies thecombination of the recommendation count C_(x) with the preferenceinformation v_(x), combination of the recommendation count C_(y) withthe preference information v_(y), and combination of the recommendationcount C_(z) with the preference information v_(z) to the serviceoffering device 63 as program recommendation-intended preferenceinformation.

In turn, the service offering device 63 references the EPG data storedin the EPG storage device 33 and thereby determines as many TV programsmatching the preference information v_(x) as the recommendation countC_(x) combined with that information v_(x) included in the programrecommendation-intended preference information coming from thepreference information processing device 62.

Likewise, the service offering device 63 determines as many TV programsmatching the preference information v_(y) as the recommendation countC_(y) combined with that information v_(y), and as many TV programsmatching the preference information v_(z) as the recommendation countC_(z) combined with that information v_(z).

The service offering device 63 then creates the recommendation screensuch as one shown in FIG. 5. The service offering device 63 recommendsthe determined TV programs by having an appropriate recommendationscreen displayed on the display device 1.

FIG. 5 is a schematic view showing a typical recommendation screen thatmay be created by the service offering device 63 according to the thirdrecommendation method.

As shown in FIG. 5, the service offering device 63 creates arecommendation screen that may list, from the top down, recommendedprogram information about as many TV programs as the recommendationcount C_(x) in keeping with the preference information v_(x),recommended program information about as many TV programs as therecommendation count C_(y) in keeping with the preference informationv_(y), and recommended program information about as many TV programs asthe recommendation count C_(z) in keeping with the preferenceinformation v_(z).

According to the above-described third recommendation method, largernumbers of TV programs are recommended in regard to the targets with thehigher operator probabilities.

The program recommendation device 15, as described above, may recommendTV programs to the target in accordance with any one of the firstthrough the third method for recommending programs.

According to the first recommendation method, one target with thehighest operator probability is selected as the operator. The preferenceinformation about the selected operator is used as programrecommendation-intended preference information, and TV programs matchingthat information are recommended. In this manner, the TV programs inkeeping with the preferences of one target with the highest operatorprobability are recommended.

According to the second recommendation method, programrecommendation-intended preference information is synthesized from thepreference information about a plurality of targets near the HD recorder3 on the basis of the operator probabilities for these operators. TVprograms are then recommended in keeping with the programrecommendation-intended preference information thus created. This makesit possible to recommend TV programs not only to the target acting asthe operator with the highest operator probability but also to othertargets in consideration of their preferences. That is, TV programs thatmay draw the attention of targets other than the single target acting asthe operator may be recommended.

According to the third recommendation method, programrecommendation-intended preference information is created by combiningthe preference information about a plurality of targets near the HDrecorder 3 with the recommendation counts corresponding to the operatorprobabilities for these targets. TV programs are then recommended inkeeping with the program recommendation-intended preference informationthus created. As with the second recommendation method, the third methodmakes it possible to recommend TV programs not only to the target actingas the operator with the highest operator probability but also to othertargets in consideration of their preferences. TV programs that maypotentially draw the attention of targets other than the single targetacting as the operator can thus be recommended.

For example, suppose that the second or the third recommendation methodis in use, that two targets are near the HD recorder 3, and that the twotargets have the same operator probability of 100 (1.0). In this case,TV programs are recommended with the preferences of the two targetstaken equally into consideration.

More specifically, according to the second recommendation method,program recommendation-intended preference information is acquired bysynthesizing the preference information about the two targets near theHD recorder 3 using equal weights. TV programs are then recommended bygiving equal consideration to the preferences of the two targets.

According to the third recommendation method, equal numbers of TVprograms matching the preferences of each of the two targets near the HDrecorder 3 are recommended.

Meanwhile, operator probabilities do not constitute ordinary nonlinearlikelihood curves. That means they are highly suitable for computingweights (ratios) used to synthesize preference information about aplurality of targets. In that respect, operator probabilities can beapplied to a wide range of synthesizing methods without being normalizedor linearized.

The use of the operator probabilities makes it possible to recommend TVprograms in consideration of the preference information about aplurality of targets and to offer other extensive services.

Although the above-described second recommendation method was shown tohave program recommendation-intended preference information synthesizedby assigning greater weights to the preference information about thetargets with the higher operator probabilities, this is not limitativeof the invention. Alternatively, program recommendation-intendedpreference information may be synthesized by assigning greater weightsto the preference information about the targets with the lower operatorprobabilities. In the latter case, TV programs are recommended in amanner drawing the attention of the users who are little interested inthe HD recorder 3.

Likewise, although the third recommendation method above was shown toget the preference information about the targets with the higheroperator probabilities to combine with larger numbers of recommended TVprograms, this is not limitative of the invention. Alternatively, thepreference information about the targets with the lower operatorprobabilities may be combined with larger numbers of recommended TVprograms. In the latter case, TV programs are also recommended in such amanner as to draw the attention of the users who are little interestedin the HD recorder 3.

Which of the first through the third recommendation methods is to beused by the program recommendation device 15 to recommend TV programsmay be determined illustratively by users' settings. It should be notedthat the methods for recommending TV programs are not limited to theabove-described first through the third method.

FIG. 6 is a block diagram showing a typical structure of the sensordevice 41 and operator probability computation device 42 in the targetinformation acquisition device 13.

The sensor device 41 is made up of a camera 91, a microphone 92, andother sensors. In operation, the sensor device 41 senses conditions andoutputs resulting sensor signals to the operator probability computationdevice 42.

More specifically, the camera 91 senses optical conditions and outputsthe resulting video data as its sensor signal. The microphone 92 sensesacoustic conditions and outputs the resulting audio data as its sensorsignal.

The sensor device 41 may optionally include an infrared sensor forsensing infrared radiation.

The operator probability computation device 42 is constituted by aninformation extraction device 101, a target information creation device102, a target information storage device 103, a probability computationdevice 104, and a model storage device 105. In operation, the operatorprobability computation device 42 acquires target information about thebehavior, gestures, etc., of targets near the HD recorder 3 on the basisof the sensor signals coming from the sensor device 41. Furthermore, theoperator probability computation device 42 computes an operatorprobability representing how likely each of the targets near the HDrecorder 3 is to be the operator on the basis of the target informationobtained from the sensor signals supplied by the sensor device 41. Theoperator probabilities thus computed are included in the targetinformation before they are output.

The information extraction device 101 is composed of an image processingdevice 111, an audio processing device 112, and other signal processingdevices for processing sensor signals. From the sensor signals suppliedby the sensor device 41, the information extraction device 101 extractsinformation about each of at least one target near the HD recorder 3 andfeeds the extracted information to the target information creationdevice 102.

The image processing device 111 is made up of a face sensor 121, amotion sensor 122, a pointing sensor 123, a sight line sensor 124, andother sensors or recognition units. In operation, the image processingdevice 111 processes video data in the sensor signals fed by the sensordevice 41, extracts from the video data a variety of information aboutthe targets near the HD recorder 3, and sends the extracted informationto the target information creation device 102.

More specifically, the face sensor 121 senses an area inside the imagecontaining the faces of targets out of the video data coming from thesensor device 41. From the sensed area in the image, the face sensor 121acquires the direction, position, and size of each face and supplieswhat is acquired to the target information creation device 102.

The face sensor 121 also recognizes the faces of the targets from thevideo data supplied by the sensor device 41. Following the facerecognition, the face sensor 121 supplies user labels identifying theusers to the target information creation device 102. The user labelsresulting from the face recognition by the face sensor 121 are usedillustratively to identify the preference information about the targetswith their faces recognized out of the preference information placed inthe preference table held by the preference table storage device 61 inthe program recommendation device 15 of FIG. 1.

From the video data supplied by the sensor device 41, the motion sensor122 extracts the direction in which each target is moving (direction ofmotion), the amount of the motion, and the position of the origin of amotion vector representing the motion of the target in question(position of motion). What is extracted is then sent to the targetinformation creation device 102.

From the video data supplied by the sensor device 41, the pointingsensor 123 senses whether a given target is making a pointing gesture.If the target is found making a pointing gesture, then the pointingsensor 123 senses the direction in which the target is pointing(pointing direction) and the position of the hand doing the pointing(pointing position) and feeds what is sensed to the target informationcreation device 102.

From the video data sent by the sensor device 41, the sight line sensor124 senses the direction of the line of sight of the target (sight linedirection) and supplies what is sensed to the target informationcreation device 102.

With ordinary interaction devices, diverse image processing algorithmsare used to extract information deemed useful for user estimation fromthe input image. In like manner, the image processing device 111 canextract diverse information from the input image.

If the camera 91 in the image processing device 111 is a so-calledstereo camera, it can extract depth information.

The audio processing device 112 processes audio data in the sensorsignals coming from the sensor device 41, extracts from the audio data avariety of information about the targets near the HD recorder 3, andsends the extracted information to the target information creationdevice 102.

More specifically, the audio processing device 112 may recognize thevoice of each target using a voice recognition unit. Following the voicerecognition, the audio processing device 112 may acquire informationabout the phonemes and rhythms of the target's voice. The audioprocessing device 112 may also obtain the direction in which the targetis talking (i.e., talking direction) using a sound source directionrecognition unit.

If the sensor device 41 is furnished with an infrared sensor asmentioned above, the information extraction device 101 includes acorresponding signal processing device. When installed, this signalprocessing device processes the output of the infrared sensor, picks uptemperatures at different portions of a given target, and feed theextracted temperature information to the target information creationdevice 102.

The sensors to be included in the sensor device 41 may be combined withany signal processing device in the information extraction device 101 aslong as the combination is capable of obtaining integral informationabout each target.

The target information creation device 102 receives information from theinformation extraction device 101 and arranges the received informationinto target information about each target.

More specifically, given the information from the information extractiondevice 101, the target information creation device 102 determines thenumber of targets near the HD recorder 3 (it also determines whetherthere is any target near the HD recorder 3). Furthermore, the targetinformation creation device 102 determines which of the targets iscovered by the information received from the information extractiondevice 101, and organizes by target the information coming from theinformation extraction device 101 to create target-specific targetinformation (i.e., target information about each of all targetsrecognized).

The target information creation device 102 creates target information atpredetermined intervals in the manner described above. The targetinformation is then supplied chronologically from the target informationcreation device 102 to the target information storage device 103.

The process of organizing information by target as discussed above maybe carried out using the so-called short term memory (STM) scheme commonto interaction devices.

There exist diverse STM techniques for organizing target-specificinformation using positional and temporal information about the inputresulting from recognition (sensing) processes. Illustratively, there isan STM technique involving the use of the closeness of sensed positionsand predicted points of motion past differential times. However, thetypes of STM techniques that may be used by the target informationcreation device 102 are not subject to particular constraints as long asthey can organize information about recognized results from each of thetargets found within the range observable by the HD recorder 3.

The target information storage device 103 temporarily stores thetarget-specific target information supplied chronologically by thetarget information creation device 102.

The probability computation device 104 computes an operator probabilityrepresenting how likely each of the targets is to be the operator usinga model stored in the model storage device 105 and in accordance withthe target information held by the target information storage device103. The operator probabilities thus computed are included in thetarget-specific target information before they are output.

More specifically, the probability computation device 104 may utilize astochastic reasoning computation model such as the Bayesian network (BN)when computing operator probabilities for each target by inputting intothe BN diverse information (called component information whereappropriate) constituting the target information stored in the targetinformation storage device 103.

Meanwhile, the Bayesian network comes in two major types: a staticBayesian network (SBN) for performing probabilistic computations usingsingle-time information, and a dynamic Bayesian network (DBN) forexecuting (reasoning) probabilistic computations using all-timeinformation preserved from the past. There are more BN ramificationscoming under the static and dynamic Bayesian networks. Any of these BNmodels may be used to compute operator probabilities.

The model for use in computing operator probabilities may be any modelas long as it suitable for reasoning the probability of a given targetbeing the operator when relevant component information is acquired outof the target-specific target information. In that respect, the model isnot limited to any one of the BN variations.

The model storage device 105 stores models for use by the probabilitycomputation device 104 in computing operator probabilities. Typically,the above-outlined BN models are retained in the model storage device105.

The processing by the image processing device 111 shown in FIG. 6 isexplained below in more detail by referring to FIGS. 7 through 9.

FIG. 7 shows a typical image taken by the camera 91.

The image of FIG. 7 shows three users as targets. More specifically, twotargets are shown sitting in the foreground while one target is shownwalking from left to right in the background.

FIGS. 8 and 9 indicate information that may be extracted from the imageof FIG. 7 by the image processing device 111.

Illustratively, from the image in FIG. 7 showing three targets, theimage processing device 111 extracts part or all of such information asthe direction of face, face position, face size, direction of motion,amount of motion, pointing direction, and direction of sight line, asdepicted in FIGS. 8 and 9.

The processing by the target information creation device 102 in FIG. 6is described below in more detail by referring to FIG. 10.

Illustratively, suppose that the items of information shown in FIGS. 8and 9 are extracted by the information extraction device 101 from theimage of FIG. 7 containing three targets and that the extractedinformation is supplied to the target information creation device 102.In this case, the target information creation device 102 determines towhich of the three targets in FIG. 7 the items of information indicatedin FIGS. 8 and 9 belong. Based on the result of the determination, thetarget information creation device 102 creates target information inwhich the items of information in FIGS. 8 and 9 are suitably organizedfor each of the three targets in FIG. 7, as shown in FIG. 10.

As described above, the target information creation device 102chronologically creates target information about each target and feedsthe created target information in chronological order to the targetinformation storage device 103 (FIG. 6).

FIG. 11 is a schematic view showing what is contained in the targetinformation storage device 103 in FIG. 6.

As shown in FIG. 11, the target information storage device 103 storesfor each of the targets involved target information with its componentscovering such items as the direction of face, face position, face size,direction of motion, amount of motion, pointing directions and directionof sight line. The target information is received chronologically fromthe target information creation device 102 by the target informationstorage device 103 for storage therein.

FIG. 12 is a schematic view showing a Bayesian network (BN) model usedby the probability computation device 104 in FIG. 6 in computingoperator probabilities. This model is stored in the model storage device105.

What is shown in FIG. 12 is a dynamic Bayesian network (DBN) model forcomputing operator probabilities (called the operator probabilitycomputation BN below where appropriate).

The operator probability computation EN in FIG. 12 has nodes thatcorrespond to component information (potentially) constituting targetinformation. For purpose of simplification and illustration, FIG. 12indicates seven nodes corresponding to a total of seven components oftarget information, including direction of face, face position, facesize, direction of motion, amount of motion, pointing direction, anddirection of sight line.

The operator probability computation BN further includes an operatornode indicating that a given target is the operator. This node, inaddition to the nodes representing the components of target information,is used to compute the probability of the target being the operator(i.e., operator probability).

As long as it has the component nodes and the operator node, theoperator probability computation BN may optionally include other nodessuch as hidden nodes or those representative of the conditions of the HDrecorder 3.

A typical node denoting the conditions of the HD recorder 3 may be onewhich indicates whether or not a DVD is loaded in the HD recorder 3.When the HD recorder 3 is not loaded with a DVD, it is unlikely for anyuser to perform operations for writing or reading data to or from theDVD. Thus the use of the node indicating whether or not a DVD is loadedin the HD recorder 3 contributes to reducing the operator probability ofany user who makes gestures for giving instructions to write or readdata to or from the DVD.

The operator probability computation BN in FIG. 12 has two Bayesiannetworks: a BN made up of component in Formation nodes and an operatornode about a time “t” (current time), and a BN constituted by componentinformation nodes and an operator node about a time “t−1,” one timeinterval earlier than the current time.

In the operator probability computation BN of FIG. 12, thick-line arrowsconnecting some nodes represent a prior network and thin-line arrowsrepresent a transition network.

In FIG. 12, the relations between the BN made of the componentinformation nodes and operator node about the time “t” on the one hand,and the BN formed by the component information nodes and operator nodeabout the time “t−1” on the other hand can also be applied to therelations between the BN composed of the component information nodes andoperator node about the time “t−1” on the one hand, and the BNconstituted by the component information nodes and operator node aboutthe time “t−2” on the other hand. For this reason, the operatorprobability computation BN in FIG. 12 may have an extended BNestablished on the left-hand side (past side) of the BN made of thecomponent information nodes and operator node about the time “t−1,” theextended BN being formed by component information nodes and an operatornode about a time “t−2.”

The operator probability computation BN in FIG. 12 may have moreextended BN's established in like manner, each extended BN being made ofsimilar component information nodes and an operator node about a fartherpast time. If the operator probability computation BN has more BN'sextended up to the one constituted by component information nodes and anoperator node about a past time “t−T,” then the component information inthe target information ranging from the time “t−T” to the current time“t” is assigned to the corresponding component information nodes, andthe operator probabilities ranging from the time “t−T” to the time “t−1”are given to the corresponding operator nodes. This makes it possible toinduce the information corresponding to the operator node about thecurrent time “t,” i.e., information representing the operatorprobability at the current time “t.”

The operator probability at the current time “t” can be computed withouthaving to feed all component information in the target informationranging from the time “t−T” to the current time “t” and all operatorprobabilities ranging from the time “t−T” to the time “t−1.” That is,with the BN model in use, the operator probability at the current time“t” can be computed using part of all component information in thetarget information ranging from the time “t−T” to the current time “t”and part of all operator probabilities ranging from the time “t−T” tothe time “t−1.”

The inference algorithms for inferring information based on the staticBN model illustratively include the Pearl πλ message passing algorithmand the junction tree algorithm whereby an exact solution can beobtained. Also included are the loopy BP and the cluster BP whereby anapproximate solution is acquired at high speed using the static BN. Theinference algorithms for inferring information based on the dynamic BN(DBN) illustratively include the 1.5 junction tree algorithm and theBoyen-Koller inference algorithm, the latter algorithm being anapplication of the former. The 1.5 junction tree algorithm permitsacquisition of an exact solution while the Boyen-Koller inferencealgorithm allows an approximate solution to be obtained at high speed.

The BN model containing the operator probability computation BN isdefined by a network structure and a parameter called a conditionalprobability table (CPT).

The network structure and the CPT defining the operator probabilitycomputation BN need to be carefully established because they cansignificantly affect the operator probabilities to be acquired by theprobability computation device 104 (FIG. 6).

Illustratively, the network structure and the CPT may be establishedmanually by designers of the HD recorder 3. Alternatively, the networkstructure and the CPT may be acquired through learning by use oflearning data prepared for the purpose.

FIG. 13 is a schematic view showing typical structures of a learningapparatus for acquiring the network structure and the conditionalprobability table (CPT) through learning.

In FIG. 13, the learning apparatus is constituted by a structurelearning device 151, a structure storage device 152, a CPT learningdevice 153, and a CPT storage device 154.

The structure learning device 151 is supplied with learning data.

The learning data is formed by combinations of target information withoperator labels, the target information being the same as what iscreated by the target information creation device 102 in FIG. 6. Thelearning data is collected as follows:

A sensor device similar to the sensor device 41 is used to sense thebehavior of a large number of people acting as targets. From the resultof the sensing, an information extraction device similar to informationextraction device 101 and a target information extraction device similarto the target information creation device 102 acquire target informationmaking up the learning data.

When the target information constituting the learning data is obtainedfrom the behavior of the targets each acting as the operator, the targetinformation is associated with target labels identifying each targetacting as the operator. If the target information is acquired from anyother behavior or gestures, the acquired information is associated withtarget labels indicating that each target is not the operator.

The structure learning device 151 supplies the component information inthe target information making up the learning data to the correspondingcomponent information nodes in the operator probability computation BN(FIG. 12), and gives the target label associated with the targetinformation to the operator node. In so doing, the structure learningdevice 151 obtains the network structure for use in the operatorprobability computation BN and feeds the obtained structure to thestructure storage device 152 for storage therein.

The structure learning device 151 supplies the CPT learning device 153with the learning data used to acquire the network structure for theoperator probability computation BN.

The CPT learning device 153 acquires the CPT based on the networkstructure stored in the structure storage device 152 and on the learningdata coming from the structure learning device 151, and supplies theacquired CPT to the CPT storage device 154 for storage therein.

The model storage device 105 in FIG. 6 stores the operator probabilitycomputation BN defined by the network structure held in the structurestorage device 152 and by the CPT kept in the CPT storage device 154.

The probability computation device 104 in FIG. 6 obtains the operatorprobability for each target using the operator probability computationBN stored in the model storage device 105 as well as the target-specifictarget information sent from the target information storage device 103.

FIG. 14 is a schematic view showing typical operator probabilitiesacquired for the three targets being imaged in FIG. 7.

In the image of FIG. 7, two targets are shown sitting in the foregroundwith one target walking in the background as discussed above. It isassumed here that one of the two targets in the foreground sitting onthe left is referred to as a target T₁, the other setting target as atarget T₂, and the target walking in the background as a target T₃. InFIG. 14, the obtained operator probability is illustratively 80 (0.80)for the target T₁, 98 (0.98) for the target T₂, and 4 (0.04) for thetarget T₃.

In the case above, the operator selection device 51 in the interactiondevice 14 of FIG. 1 selects the target T₂ as the operator having thehighest operator probability of 98 out of the three targets T₁, T₂, andT₃. The user can thus become the operator without having to performcomplicated actions such as saying “I am the user.”

In other words, where there are at least two users who can be targets,there is no need for them to carry out elaborate actions for switchingthe role of the operator. The procedure of one user taking over fromanother as the new operator is thus effected in seamless fashion.

In the above example, the interaction device 14 was shown selecting thetarget with the highest operator probability as the operator.Alternatively, at the design stage of the HD recorder 3, it is possibleto establish a threshold level against which the operator probability ofa given target potentially becoming the operator may be checked. In thiscase, the interaction device 14 may select as the operator any targethaving an operator probability exceeding the established thresholdlevel. The interaction device 14 may then select the action to beperformed by the HD recorder 3 depending on the number of targetsselected as the operator each.

Furthermore, since operator probabilities are susceptible to computingratios, the HD recorder 3 may take advantage of such ratios in offeringflexible services.

Illustratively, the program recommendation device 15 in FIG. 15 may setthe operator probability threshold to 50. In this case, every targetwith an operator probability in excess of that threshold level isselected as the operator so that TV programs in consideration of thepreference of the selected operator may be recommended.

More specifically, as shown in FIG. 14, if the operator probability is80 for the target T₁, 98 for the target T₂ and 4 for the target T₃, andif the threshold level is set to 50, then the program recommendationdevice 15 selects as operators the targets T₁ and T₂ with their operatorprobabilities exceeding the threshold (50).

Furthermore, the program recommendation device 15 synthesizes thepreference information about the target T₁ and the preferenceinformation about the target T₂ using the ratio of 80 to 98 based on theoperator probability of 80 for the target T₁ and 98 for the target T₂.The program recommendation device 15 then recommends TV programs inkeeping with program recommendation-intended preference informationderived from the synthesis. In this case, it is possible to recommend TVprograms that are more in line with the current situation (i.e., TVprograms matching the preferences of the targets T₁ and T₂ manifestingtheir willingness to become the operators).

Described below in reference to the flowchart of FIG. 15 is a processperformed by the operator probability computation device 42 in FIG. 6.

The sensor device 41 outputs sensor signals at predetermined intervalsto the operator probability computation device 42. With the sensorsignals sent from the sensor device 41, the operator probabilitycomputation device 42 receives the signals and forwards them to theinformation extraction device 101.

In step S101, the information extraction device 101 processes the sensorsignals from the sensor device 41 and extracts information about thetargets near the HD recorder 3 from the processed signals. The extractedinformation is sent to the target information creation device 102,before step S102 is reached.

In step S102, the target information creation device 102 createstarget-specific target information illustratively through STM out of theinformation supplied by the information extraction device 101. Thecreated target information is fed to the target information storagedevice 103 for storage therein, before step S103 is reached.

In step S103, the probability computation device 104 computes theoperator probability for each target by feeding (inputting) thetarget-specific target information held by the target informationstorage device 103 into the operator probability computation BN storedin the model storage device 105. Step S103 is followed by step S104.

In step S104, the probability computation device 104 causes thetarget-specific operator probability obtained in step S103 to beincluded in the target information about the respective targets, beforeoutputting the target-specific target information together with theoperator probabilities.

Thereafter, every time new sensor signals are output illustratively fromthe sensor device 41, steps S101 through S104 are carried out.

The processing by the HD recorder 3 in FIG. 1 will now be outlined byreferring to FIG. 16.

In step S201, the operator probability computation device 42 extractsinformation (e.g., direction of face and other items) about each of thetargets near the HD recorder 3 from the sensor signals output by thesensor device 41, and organizes the extracted information into targetinformation about each target.

In step S202, the operator probability computation device 42 computesthe operator probability for each target by feeding the target-specifictarget information into the operator probability computation BN. Theoperator probability computation device 12 causes the operatorprobability thus computed for each target to be included in the targetinformation about the respective targets, before outputting thetarget-specific target information together with the operatorprobabilities.

In step S203, the interaction device 14 illustratively selects as theoperator the target having the highest operator probability contained inthe target-specific target information output by the operatorprobability computation device 42. The interaction device 14 proceeds toextract the target information about the operator from thetarget-specific target information output by the operator probabilitycomputation device 42.

In step S204, the interaction device 14 estimates the scene of theoperator (i.e., scene in which the operator behaved in a particularmanner) based on the target information about the operator. Inaccordance with the result of the estimation, the interaction device 14determines the action to be performed by the HD recorder 3. Theinteraction device 14 proceeds to make arrangements for the action to becarried out.

The series of steps or processes performed by the above-describedoperator probability computation device 42, interaction device 14, andprogram recommendation device 15 may be executed either by hardware orby software. For the software-based processing to take place, theprograms constituting the software may be either incorporated beforehandin dedicated hardware of a computer for program execution or installedupon use into a general-purpose personal computer or like equipmentcapable of executing diverse functions based on the installed programs.

FIG. 17 is a block diagram showing a typical structure of a computeraccording to the embodiment of the present invention. The computer hasrelevant programs installed so as to execute the above-described stepsor processes.

The programs may be written beforehand to a hard disk drive 205 or a ROM203 incorporated in the computer as a recording medium.

Alternatively, the programs may be stored temporarily or permanently ona removable recording medium 211 such as flexible disks, CD-ROM (CompactDisc Read-Only Memory), MO (Magneto-Optical) disks, DVD (DigitalVersatile Disc), magnetic disks, or a semiconductor memory. Theremovable recording medium 211 then may be offered as a softwarepackage.

Instead of getting installed from the removable recording medium 211,the programs may be transferred from download sites to the computerwirelessly via digital satellite broadcast links or in wired fashionover a network such as the Internet. The computer may receive thetransferred programs through a communication device 208 and have thereceived programs installed onto the hard disk drive 205 inside.

The computer incorporates a CPU (Central Processing Unit) 202. Aninput/output interface 210 is connected to the CPU 202 via a bus 201.The user may enter commands into the CPU 202 by operating an inputdevice 207 made up of a keyboard, a mouse, a microphones etc. Given suchcommands, the CPU 202 executes relevant programs stored in the ROM 203.For program execution, the CPU 203 may alternatively load into a RAM(Random Access Memory) 204 the programs kept on the hard disk drive 205,the programs transferred via satellite or over the network and installedonto the hard disk drive 205 upon receipt by the communication device208, or the programs retrieved from the removable recording medium 211set in a drive 209 and installed onto the hard disk drive 205. Byexecuting the programs, the CPU 202 may carry out the steps or processesin a manner shown in the above-described flowcharts or as illustrated inthe block diagrams above. The result of the processing may be sent bythe CPU 202 to an output device 206 constituted by an LCD (LiquidCrystal Display), speakers, etc., for output to the outside through theinput/output interface 210; to the communication device 208 fortransmission to the outside; or to the hard disk drive 205 for storagethereon.

In this specification, the steps describing the programs for causing thecomputer to execute diverse processing represent not only the processesthat are to be carried out in the sequence depicted in the flowcharts(i.e., on a time series basis) but also processes that may be performedparallelly or individually and not chronologically (e.g., in parallel orobject-oriented fashion).

The programs may be processed by a single computer or by a plurality ofcomputers on a distributed basis. The programs may also be transferredto a remote computer or computers for execution.

Although the embodiment of the present invention has been described asapplicable to the AV system, this is not limitative of the embodiment ofthe invention. Alternatively, the embodiment of the invention may beapplied to diverse interaction apparatuses such as automatic telephoneanswering systems that handle voice signals as input, ticket reservationsystems that deal with entries made through touch-sensitive panels, andTV program reservation systems that accept images as input.

The devices or sections that carry out relevant processes based onoperator probabilities are not limited to the interaction device 14 andprogram recommendation device 15 shown in FIG. 1.

The interaction device 14 (FIG. 1) may cause the output control device34 to display on the display device 1 a user label identifying thetarget (user) selected as the operator by the operator selection device51. In this case, the targets near the HD recorder 3 can readilyrecognize who the operator is at the moment.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factor in so far as they arewithin the scope of the appended claims or the equivalents thereof.

1. An information processing apparatus for carrying out processes, saidinformation processing apparatus comprising: a sensor device configuredto sense conditions including information about a plurality of targetseach being one of a plurality of users staying close to the apparatus;an operator probability computation device configured to compute aplurality of operator probabilities each corresponding to one of theplurality of targets on the basis of the information about the pluralityof targets, each of the plurality of operator probability representinghow likely the corresponding target is to be an operator operating saidinformation processing apparatus such that the information processingapparatus carries out processes in response to how the operator behaves;an operator selection device configured to select a selected operatorbased on the plurality of operator probabilities; an action selectiondevice configured to select an action to be performed based onestimating behavior of the selected operator in one or more scenessensed by the sensor device; and an action execution device configuredto perform a predetermined process based on the selected action.
 2. Theinformation processing apparatus according to claim 1, wherein saidoperator probability computation device further comprises: informationextraction means configured to extract from output of said sensordevice, information about each of the plurality of targets; targetinformation creation means configured to create target information abouteach of the plurality of targets out of the information extracted bysaid information extraction means; and probability computation meansconfigured to compute said plurality of operator pluralities on thebasis of the target information about each of the plurality of targets.3. The information processing apparatus according to claim 2, whereinsaid probability computation device further comprises storage meansconfigured to store the target information created chronologically bysaid target information creation means; wherein said probabilitycomputation means computes said plurality of operator pluralities basedon the chronological target information stored in said storage means. 4.The information processing apparatus according to claim 1, wherein saidoperator probability computation device computes said plurality ofoperator pluralities using a Bayesian network.
 5. The informationprocessing apparatus according to claim 1, wherein the operatorselection device is configured to select out of the plurality of targetsa target with the highest corresponding operator probability as theselected operator.
 6. The information processing apparatus according toclaim 1 further comprising: preference information storage meansconfigured to store preference information representative of preferencesof each of the plurality of targets; preference information processingmeans configured to create program recommendation-intended preferenceinformation for recommending programs out of the preference informationabout each of the plurality of targets in accordance with said pluralityof operator probabilities; and service offering means configured tooffer a program recommending service based on said programrecommendation-intended preference information.
 7. The informationprocessing apparatus according to claim 6, wherein said preferenceinformation processing means creates as said programrecommendation-intended preference information, the informationsynthesizing the preference information about each of the plurality oftargets on the basis of said operator probability for each of theplurality of targets.
 8. The information processing apparatus accordingto claim 6, wherein said preference information creation means createsas said program recommendation-intended preference information, theinformation associating the preference information about each of theplurality of targets with the number of programs to be recommended basedon said preference information, on the basis of said operatorprobability for each of the plurality of targets.
 9. An informationprocessing method for causing an information processing apparatus tocarry out a predetermined process, the method comprising the steps of:receiving information about a plurality of targets each being one of aplurality of users staying close to the apparatus; computing, on thebasis of the information about the plurality of targets, a plurality ofoperator probabilities each corresponding to one of the plurality oftargets and each representing how likely the corresponding target is tobe an operator operating said information processing apparatus such thatthe information processing apparatus carries out processes in responseto how the operator behaves; selecting a selected operator based on theplurality of operator probabilities; selecting an action to be performedbased on estimating behavior of the selected operator in one or morescenes sensed by the sensor device; and performing the predeterminedprocess based on the selected action.
 10. A non-transitory computerreadable medium encoded with a computer program product, the computerprogram product causing a computer to carry out an informationprocessing procedure for an information processing apparatus, theprocedure comprising the steps of: receiving information about aplurality of targets each being one of a plurality of users stayingclose to the apparatus; computing, on the basis of the information aboutthe plurality of targets, a plurality of operator probabilities eachcorresponding to one of the plurality of targets a plurality of operatorprobabilities each corresponding to one of the plurality of targets andeach representing how likely the corresponding target is to be anoperator operating said information processing apparatus such that theinformation processing apparatus carries out processes in response tohow the operator behaves; selecting a selected operator based on theplurality of operator probabilities; selecting an action to be performedbased on estimating behavior of the selected operator in one or morescenes sensed by the sensor device; and performing a predeterminedprocess based on the selected action.
 11. The information processingmethod of claim 9, wherein the computing a plurality of operatorprobabilities includes: extracting, from output of the sensor device,information about each of the plurality of targets; creating targetinformation about each of the plurality of targets based on theextracted information; and computing, based on the target informationabout each of the plurality of targets, the operator probability foreach of the one or more targets.
 12. The information processing methodof claim 11, wherein the computing a plurality of operator probabilitiesfurther includes storing the target information created chronologically,and computing the plurality of operator probabilities based on thestored chronological target information.
 13. The information processingmethod of claim 9, wherein the plurality of operator probabilities arecomputed using Bayesian network.
 14. The information processing methodof claim 9 wherein selecting a selected operator comprises selecting,out of the plurality of targets, a target with the highest correspondingoperator probability as the selected operator.
 15. The informationprocessing method of claim 9, further comprising: storing preferenceinformation representative of preferences of each of the plurality oftargets; creating program recommendation-intended preference informationfor recommending programs out of the preference information about eachof the plurality of targets in accordance with the plurality of operatorprobabilities; and offering a program recommending service based on theprogram recommendation-intended preference information.
 16. Theinformation processing method of claim 15, wherein the creating theprogram recommendation-intended preference information includes creatinginformation synthesizing the preference information about each of theplurality of targets on the basis of the operator probability for eachof the plurality of targets.
 17. The information processing method ofclaim 15, wherein the creating the program recommendation-intendedpreference information includes creating information associating thepreference information about each of the plurality of targets with thenumber of programs to be recommended based on said preferenceinformation, on the basis of said operator probability for each of theplurality of targets.
 18. The information processing method of claim 1,wherein the plurality of operator probabilities is based on actions ofthe plurality of targets sensed by the sensor device.
 19. Theinformation processing apparatus of claim 1, wherein the operatoroperates a television through operating the information processingapparatus.
 20. The information processing method of claim 9, whereincomputing the plurality of operator probabilities is based on actions ofthe plurality of targets sensed by the sensor device.
 21. Theinformation processing method of claim 9, wherein the operator operatesa television through operating the information processing apparatus. 22.The information processing apparatus of claim 1, wherein the informationabout the plurality of targets includes one or more of motion direction,motion amount, pointing direction, hand position, line of sightdirection, talking direction, face direction, face position, and facesize.
 23. The information processing method of claim 9, wherein theinformation about the plurality of targets includes one or more ofmotion direction, motion amount, pointing direction, hand position, lineof sight direction, talking direction, face direction, face position,and face size.